Aspects of this disclosure are generally related to data storage systems which can be used to maintain relatively large data sets and support a relatively large number of concurrent users. The basic building blocks of a data storage system may include computing nodes such as storage engines or storage servers. The computing nodes provide access to data storage devices such as disk drives and flash drives. A plurality of computing nodes can be interconnected in order to function together as a cluster. A plurality of clusters can be interconnected by communication links to enable the clusters to function as a single distributed storage system.